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ABSTRACT 

We address the problem of managing historical data for large evolv- 
ing information networks like social networks or citation networks, 
with the goal to enable temporal and evolutionary queries and anal- 
ysis. We present the design and architecture of a distributed graph 
database system that stores the entire history of a network and pro- 
vides support for efficient retrieval of multiple graphs from arbi- 
trary time points in the past, in addition to maintaining the current 
state for ongoing updates. Our system exposes a general program- 
matic API to process and analyze the retrieved snapshots. We in- 
troduce DeltaGraph, a novel, extensible, highly tunable, and dis- 
tributed hierarchical index structure that enables compactly record- 
ing the historical information, and that supports efficient retrieval 
of historical graph snapshots for single-site or parallel processing. 
Along with the original graph data, DeltaGraph can also maintain 
and index auxiliary information; this functionality can be used to 
extend the structure to efficiently execute queries like subgraph pat- 
tern matching over historical data. We develop analytical models 
for both the storage space needed and the snapshot retrieval times 
to aid in choosing the right parameters for a specific scenario. In 
addition, we present strategies for materializing portions of the 
historical graph state in memory to further speed up the retrieval 
process. Secondly, we present an in-memory graph data structure 
called GraphPool that can maintain hundreds of historical graph 
instances in main memory in a non-redundant manner. We present 
a comprehensive experimental evaluation that illustrates the effec- 
tiveness of our proposed techniques at managing historical graph 
information. 

1. INTRODUCTION 

In recent years, we have witnessed an increasing abundance of 
observational data describing various types of information networks, 
including social networks, biological networks, citation networks, 
financial transaction networks, communication networks, to name a 
few. There is much work on analyzing such networks to understand 
various social and natural phenomena like: "how the entities in a 
network interact", "how information spreads", "what are the most 
important (central) entities", and "what are the key building blocks 
of a network" . With the increasing availability of the digital trace of 
such networks over time, the topic of network analysis has naturally 
extended its scope to temporal analysis of networks, which has the 
potential to lend much better insights into various phenomena, es- 
pecially those relating to the temporal or evolutionary aspects of 
the network. For example, we may want to know: "which ana- 
lytical model best captures network evolution", "how information 
spreads over time", "who are the people with the steepest increase 
in centrality measures over a period of time", "what is the aver- 
age monthly density of the network since 1997", "how the clusters 
in the network evolve over time" etc. Historical queries like, "who 



had the highest PageRank centrality in a citation network in 1960", 
"which year amongst 2001 and 2004 had the smallest network di- 
ameter", "how many new triangles have been formed in the net- 
work over the last year", also involve the temporal aspect of the 
network. More generally a network analyst may want to process 
the historical trace of a network in different, usually unpredictable, 
ways to gain insights into various phenomena. There is also interest 
in visualizations of such networks over time (2). 

To support a broad range of network analysis tasks, we require 
a grapqjdata management system at the backend capable of low- 
cost storage and efficient retrieval of the historical network infor- 
mation, in addition to maintaining the current state of the network 
for updates and other queries, temporal or otherwise. However, the 
existing solutions for graph data management lack adequate tech- 
niques for temporal annotation, or for storage and retrieval of large 
scale historical changes on the graph. In this paper, we present the 
design of a graph data management system that we are building to 
provide support for executing temporal analysis queries and histor- 
ical queries over large-scale evolving information networks. 

Our focus in this paper is on supporting snapshot queries where 
the goal is to retrieve in memory one or more historical snapshots 
of the information network as of specified time points. The typi- 
cally unpredictable and non-declarative nature of network analysis 
makes this perhaps the most important type of query that needs 
to be supported by such a system. Furthermore, the snapshot re- 
trieval times must be sufficiently low so that the analysis can be 
performed in an interactive manner (this is especially vital for visu- 
alization tools). There is a large body of work in temporal relational 
databases that has attempted to address some of these challenges 
for relational data (we discuss the related work in more detail in 
the next section). However our primary focus on efficiently re- 
trieving possibly 100's of snapshots in memory, while maintaining 
the current state of the database for ongoing updates and queries, 
required us to rethink several key design decisions and data struc- 
tures in such a system. We assume there is enough memory to hold 
the retrieved snapshots in memory (we discuss below how we ex- 
ploit overlap in the retrieved snapshots to minimize the memory 
requirements); we allow the snapshots to be retrieved in a parti- 
tioned fashion across a set of machines in parallel to handle very 
large scale networks. This design decision was motivated by both 
the current hardware trends and the fact that, most network analysis 
tasks tend to access the underlying network in unpredictable ways, 
leading to unacceptably high penalties if the data does not fit in 
memory. Most current large-scale graph analysis systems, includ- 
ing Pregel [18], Giraplf] Trinity [29], Cassovary (Twitter graph 
library |j Pegasus [13], load the entire graph into memory prior to 

1 We use the terms graph and network interchangeably in this paper. 
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Figure 1: Dynamic network analysis (e.g., understanding how 
"communities" evolve in a social network, how centrality scores 
change in co-authorship networks, etc.) can lend important in- 
sights into social, cultural, and natural phenomena. The right 
plot was constructed using our system over the DBLP network, 
and shows the evolution of the nodes ranked in top 25 in 2004. 



execution. 

The cornerstone of our system is a novel hierarchical index struc- 
ture, called DeltaGraph, over the historical trace of a network. A 
DeltaGraph is a rooted, directed graph whose lowest level corre- 
sponds to the snapshots of the network over time (that are not ex- 
plicitly stored), and the interior nodes correspond to graphs con- 
structed by combining the lower level graphs (these are typically 
not valid graphs as of any specific time point). The information 
stored with each edge, called edge deltas, is sufficient to construct 
the graph corresponding to the target node from the graph corre- 
sponding to the source node, and thus a specific snapshot can be 
created by traversing any path from the root to the snapshot. While 
conceptually simple, DeltaGraph is a very powerful, extensible, 
general, and tunable index structure that enables trading off the 
different resources and user requirements as per a specific applica- 
tion's need. By appropriately tuning the DeltaGraph construction 
parameters, we can trade decreased snapshot retrieval times for in- 
creased disk storage requirements. One key parameter of the Delt- 
aGraph enables us to control the distribution of average snapshot 
retrieval times over history. Portions of the DeltaGraph can be pre- 
fetched and materialized, allowing us to trade increased memory 
utilization for reduced query times. Many of these decisions can 
be made at run- time, enabling us to adaptively respond to chang- 
ing resource characteristics or user requirements. One immediate 
consequence is also that we can optimally use the "current graph" 
that is in memory at all times to reduce the snapshot retrieval times. 
DeltaGraph is highly extensible, providing a user the opportunity 
to define additional indexes to be stored on edge deltas in order to 
perform specific operations more efficiently. Finally, DeltaGraph 
utilizes several other optimizations, including a column-oriented 
storage to minimize the amount of data that needs to be fetched 
to answer a query, and multi-query optimization to simultaneously 
retrieve many snapshots. 

DeltaGraph naturally enables distributed storage and processing 
to scale to very large graphs. The edge deltas can be stored in a 
distributed fashion through use of horizontal partitioning, and the 
historical snapshots can be loaded parallely onto a set of machines 
in a partitioned fashion; in general, the two partitionings need not 
be aligned, but for computational efficiency, we currently require 
that they be aligned. Horizontal partitioning also results in lower 



snapshot retrieval latencies since the different deltas needed for re- 
construction can be fetched in parallel. 

The second key component of our system is an in-memory data 
structure called GraphPool. A typical network evolution query 
may require analyzing 100's of snapshots from the history of a 
graph. Maintaining these snapshots in memory independently of 
each other would likely be infeasible. The GraphPool data structure 
exploits the commonalities in the snapshots that are currently in 
memory, by overlaying them on a single graph data structure (typi- 
cally a union of all the snapshots in memory). GraphPool also em- 
ploys several optimizations to minimize the amount of work needed 
to incorporate a new snapshot and to clean up when a snapshot is 
purged after the analysis has completed. We have implemented the 
GraphPool as a completely in-memory structure because of the in- 
efficiency of general graph algorithms on a disk-resident/buffered- 
memory storage scheme. However, the design of our system itself 
enforces no such restriction on GraphPool. We address the general 
problem of scalability through horizontal partitioning, as explained 
later. 

We have built a prototype implementation of our system in Java, 
using the Kyoto Cabinejj disk-based key-value store as the back- 
end engine to store the DeltaGraph; in the distributed case, we run 
one instance on each machine. Our design decision to use a key- 
value store at the back-end was motivated by the flexibility, the 
fast retrieval times, and the scalability afforded by such systems; 
since we only require a simple get/put interface from the storage 
engine, we can easily plug in other cloud-based, distributed key- 
value stores like HBas^Jor Cassandra [15]. Our comprehensive 
experimental evaluation shows that our system can retrieve histori- 
cal snapshots containing up to millions of nodes and edges in sev- 
eral 100's of milliseconds or less, often an order of magnitude faster 
than prior techniques like interval trees, and that the execution time 
penalties of our in-memory data structure are minimal. 

Finally, we note that our proposed techniques are general and 
can be used for efficient snapshot retrieval in temporal relational 
databases as well. In fact, both the DeltaGraph and the GraphPool 
data structures treat the network as a collection of objects and do 
not exploit any properties of the graphical structure of the data. 

Outline: We begin with a discussion of the prior work (Section|2|. 
We then discuss the key components of the system, the data model, 
and present the high level system architecture (Section[3]). Then, we 
describe the DeltaGraph structure in detail (Section[4|, and develop 
analytical models for the storage space and snapshot access times 
(Section[5]). We then briefly discuss GraphPool (Section[6]). Finally, 
we present the results of our experimental evaluation (Section [7]). 

2. RELATED WORK 

There has been an increasing interest in dynamic network anal- 
ysis over the last decade, fueled by the increasing availability of 
large volumes of temporally annotated network data. Many works 
have focused on designing analytical models that capture how a 
network evolves, with a primary focus on social networks and the 
Web (see, e.g., fT] |17|[T4) ). There is also much work on under- 
standing how communities evolve, identifying key individuals, and 
locating hidden groups in dynamic networks. Berger-Wolf et al. [5 , 
[35) , Tang et al. {33 1 and Greene et al. [ 1 1 1 address the problem 
of community evolution in dynamic networks. McCulloh and Car- 
ley (20| present techniques for social change detection. Asur et 
al. (4]| present a framework for characterizing the complex behav- 
ioral patterns of individuals and communities over time. In a recent 
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work, Ahn et al. (2) present an exhaustive taxonomy of temporal vi- 
sualization tasks. Biologists are interested in discovering historical 
events leading to a known state of a Biological network (e.g., [22]). 
Ren et al. [25] analyze evolution of shortest paths between a pair 
of vertices over a set of snapshots from the history. Our goal in 
this work is to build a graph data management system that can effi- 
ciently and scalably support these types of dynamic network anal- 
ysis tasks over large volumes of data in real-time. 

There is a vast body of literature on temporal relational databases, 
starting with the early work in the 80' s on developing temporal data 
models and temporal query languages. We won't attempt to present 
a exhaustive survey of that work, but instead refer the reader to sev- 
eral surveys and books on this topic (7] [3T] [23] [34] [9] [32] |27) . The 
most basic concepts that a relational temporal database is based 
upon are valid time and transaction time, considered orthogonal 
to each other. Valid time denotes the time period during which a 
fact is true with respect to the real world. Transaction time is the 
time when a fact is stored in the database. A valid-time tempo- 
ral database permits correction of a previously incorrectly stored 
fact [31], unlike trans action- time databases where an inquiry into 
the past may yield the previously known, perhaps incorrect version 
of a fact. Under this nomenclature, our data management system is 
based on valid time - for us the time the information was entered 
in the database is not critical, but our focus is on the time period 
when the information is true. 

From a querying perspective, both valid-time and transaction- 
time databases can be treated as simply collections of intervals (27), 
however indexing structures that assume transaction times can of- 
ten be simpler since they don't need to support arbitrary inserts or 
deletes into the index. Salzberg and Tsotras [ 27] present a compre- 
hensive survey of indexing structures for temporal databases. They 
also present a classification of different queries that one may ask 
over a temporal database. Under their notation, our focus in this 
work is on the valid timeslice query, where the goal is to retrieve 
all the entities and their attribute values that are valid as of a spe- 
cific time point. We discuss the related work on snapshot retrieval 
queries in more detail in Section [TT] 

There has been resurgence of interest in general-purpose graph 
data management systems in both academia and industry. Several 
commercial and open- source graph management systems are be- 
ing actively developed (e.g., Neo4j^] GBas j 7 ] Pr egel [ijj, Giraph, 
Trinity (29], Cassovary, Pegasus (13] , GPS |26|). There is much 
ongoing work on efficient techniques for answering various types 
of queries over graphs and on building indexing structures for them. 
However, we are not aware of any graph data management system 
that focuses on optimizing snapshot retrieval queries over historical 
traces, and on supporting rich temporal analysis of large networks. 

There is also prior work on temporal RDF data and temporal 
XML Data. Motik [ 21] presents a logic-based approach to rep- 
resenting valid time in RDF and OWL. Several works (e.g., |24| 
36 ]) have considered the problems of subgraph pattern matching or 
SPARQL query evaluation over temporally annotated RDF data. 
There is also much work on version management in XML data 
stores. Most scientific datasets are semistructured in nature and can 
be effectively represented in XML format [8 ]. Lam and Wong [16] 
use complete deltas, which can be traversed in either direction of 
time for efficient retrieval. Other systems store the current version 
as a snapshot and the historical versions as deltas from the cur- 
rent version (19] . For such a system, the deltas only need to be 
unidirectional. Ghandeharizadeh et al. 1 10] provide a formalism 
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on deltas, which includes a delta arithmetic. All these approaches 
assume unique node identifiers to merge deltas with deltas or snap- 
shots. Buneman et al. [8] propose merging all the versions of the 
database into one single hierarchical data structure for efficient re- 
trieval. In a recent work, Seering et al. (28] presented a disk based 
versioning system using efficient delta encoding to minimize space 
consumption and retrieval time in array-based systems. However, 
none of that prior work focuses on snapshot retrieval in general 
graph databases, or proposes techniques that can flexibly exploit 
the memory-resident information. 

3. SYSTEM OVERVIEW 

We begin with briefly describing our graph data model and cate- 
gorizing the types of changes that it may capture. We then discuss 
different types of snapshot retrieval queries that we support in our 
system, followed by the key components of the system architecture. 

3.1 Graph Data Model 

The most basic model of a graph over a period of time is as a 
collection of graph snapshots, one corresponding to each time in- 
stance (we assume discrete time). Each such graph snapshot con- 
tains a set of nodes and a set of edges. The nodes and edges are 
assigned unique ids at the time of their creation, which are not re- 
assigned after deletion of the components (a deletion followed by a 
re-insertion results in assignment of a new id). A node or an edge 
may be associated with a list of attribute- value pairs; the list of at- 
tribute names is not fixed a priori and new attributes may be added 
at any time. Additionally an edge contains the information about 
whether it is a directed edge or an undirected edge. 

We define an event as the record of an atomic activity in the net- 
work. An event could pertain to either the creation or deletion of 
an edge or node, or change in an attribute value of a node or an 
edge. Alternatively, an event can express the occurrence of a tran- 
sient edge or node that is valid only for that time instance instead 
of an interval (e.g., a "message" from a node to another node). Be- 
ing atomic refers to the fact that the activity can not be logically 
broken down further into smaller events. Hence, an event always 
corresponds to a single timepoint. So, the valid time interval of an 
edge, [t s ,t e ], is expressed by two different events, edge addition 
and deletion events at t s and t e respectively. The exact contents of 
an event depend on the event type; below we show examples of a 
new edge event (NE), and an update node attribute event (UNA). 

(a) {NE, N:23, N:4590, directed : no, 11/29/03 10:10} 

(b) {UNA, N:23, 'job', old:'..', new:\.', 11/29/07 17:00} 

We treat events as bidirectional, i.e., they could be applied to a 
database snapshot in either direction of time. For example, say that 
at times tk-i and the graph snapshots are Gk-i and Gk respec- 
tively. If E is the set of all events at time tfc, we have that: 

Gk — Gk-i + E. Gk-i — Gk — E 

where the + and — operators denote application of the events in E 
in the forward and the backward direction. All events are recorded 
in the direction of evolving time, i.e., going ahead in time. A list of 
chronologically organized events is called an eventlist. 

3.2 System Overview 

Figure [2] shows a high level overview of our system and its key 
components. At a high level, there are multiple ways that a user or 
an application may interact with a historical graph database. Given 
the wide variety of network analysis or visualization tasks that are 
commonly executed against an information network, we expect a 
large fraction of these interactions will be through a programmatic 
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Figure 2: Architecture of our system: our focus in this work is 
on the components below the dashed line. 



API where the user or the application programmer writes her own 
code to operate on the graph (as shown in the figure). Such interac- 
tions result in what we call snapshot queries being executed against 
the database system. Executing such queries is the primary focus 
of this paper, and we further discuss these types of queries below. 
In ongoing work, we are also working on developing a high-level 
declarative query language (similar to TSQL [32]) and query pro- 
cessing techniques to execute such queries against our database. As 
a concrete example, an analyst who may have designed a new net- 
work evolution model and wants to see how it fits the observed data, 
may want to retrieve a set of historical snapshots and process them 
using the programmatic API. On the other hand, a declarative query 
language may better fit the needs of a user interested in searching 
for a temporal pattern (e.g., find nodes that had the fastest growth 
in the number of neighbors since joining the network). 

Next, we briefly discuss snapshot queries and the key compo- 
nents of the system. 

3.2.1 Snapshot Queries 

We differentiate between a singlepoint snapshot query and a mul- 
tipoint snapshot query. An example of the first query is: "Retrieve 
the graph as of January 2, 1995". On the other hand, a multipoint 
snapshot query requires us to simultaneously retrieve multiple his- 
torical snapshots (e.g, "Retrieve the graphs as of every Sunday be- 
tween 1994 to 2004"). We also support more complex snapshot 
queries where a TimeExpression or a time interval is specified in- 
stead. Any snapshot query can specify whether it requires only the 
structure of the graph, or a specified subset of the node or edge 
attributes, or all attributes. 

Specifically, the following is a list of some of the retrieval func- 
tions that we support in our programmatic API. 
GetHistGraph(Time t, String attr .options): In the basic single- 
point graph retrieval call, the first parameter indicates the time; 
the second parameter indicates the attribute information to be 
fetched as a string formed by concatenating sub-options listed 
in Table [T] For example, to specify that all node attributes ex- 



Option 


Explanation 


-node: all (default) 


None of the node attributes 


+node:all 


All node attributes 


+node:attrl 


Node attribute named "attrl"; overrides 
"-node: all" for that attribute 


-node:attrl 


Node attribute named "attrl"; overrides 
"+node:all" for that attribute 



Table 1: Options for node attribute retrieval. Similar options 
exist for edge attribute retrieval. 



cept salary, and the edge attribute name should be fetched, we 
would use: attr_options = "+node:all-node:salary+edge:name". 

GetHistGraphs(List<Time> tJist, String attr_options), where 
tJist specifies a list of time points. 

GetHistGraph(TimeExpression tex, String attr_options): This is 
used to retrieve a hypothetical graph using a multinomial Boolean 
expression over time points. For example, the expression (£i A 
—1^2) specifies the components of the graph that were valid at 
time t\ but not at time £2. The TimeExpression data struc- 
ture consists of a list of k time points, {£1, £2, • • • , £fc}, and a 
Boolean expression over them. 

GetHistGraphInterval(Time t S9 Time £ e , String attr_options): 

This is used to retrieve a graph over all the elements that were 
added during the time interval [£ s , £ e ). It also fetches the tran- 
sient events, not fetched (by definition) by the above calls. 
The (Java) code snippet below shows an example program that re- 
trieves several graphs, and operates upon them. 



/* Loading the index */ 

GraphManager gm = new GraphManager(. . . ); 

gm.loadDeltaGraphIndex(. . . ); 

/* Retrieve the historical graph structure along with node names as of 
Jan 2, 1985 */ 

HistGraph hi = gm.GetHistGraph(" 1/2/1 985", "+node:name"); 

/* Traversing the graph*/ 

List<HistNode> nodes = hl.getNodes(); 

List<HistNode> neighborList = nodes. get(0).getNeighbors(); 

HistEdge ed = hl.getEdgeObj(nodes.get(0), neighborList. get(0)); 

/* Retrieve the historical graph structure alone on Jan 2, 1986 and Jan 
2, 1987 */ 

listOfDates.add("l/2/1986"); 
listOfDates.add("l/2/1987"); 

List<HistGraph> hi = gm.getHistGraphs(listOfDates, ""); 



Eventually, our goal is to support Blueprints, a collection of inter- 
faces analogous to JDBC but for graph data (we currently support 
a subset). Blueprints is a generic graph Java API that already binds 
to various graph database backends (e.g., Neo4j), and many graph 
processing and programming frameworks are built on top of it (e.g., 
Gremlin, a graph traversal languag^J Furnace, a graph algorithms 
packag^j etc.). By supporting the Blueprints API, we immediately 
enable use of many of these already existing toolkits. 

3.2.2 Key Components 

There are two key data structure components of our system. 
1 . GraphPool is an in-memory data structure that can store multi- 
ple graphs together in a compact way by overlaying the graphs 
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on top of each other. At any time, the GraphPool contains: (1) 
the current graph that reflects the current state of the network, 
(2) the historical snapshots, retrieved from the past using the 
commands above and possibly modified by an application pro- 
gram, and (3) materialized graphs, which are graphs that corre- 
spond interior or leaf nodes in the DeltaGraph, but may not cor- 
respond to any valid graph snapshot (Section [43}. GraphPool 
exploits redundancy amongst the different graph snapshots that 
need to be retrieved, and considerably reduces the memory re- 
quirements for historical queries. We discuss GraphPool in de- 
tail in Section |6] 

2. DeltaGraph is an index structure that stores the historical net- 
work data using a hierarchical index structure over deltas and 
leaf-level eventlists (called leaf-eventlists). To execute a snap- 
shot retrieval query, a set of appropriate deltas and leaf-eventlists 
is fetched and the resulting graph snapshot is overlaid on the 
existing set of graphs in the GraphPool. The structure of the 
DeltaGraph itself, called DeltaGraph skeleton, is maintained 
as a weighted graph in memory (it contains statistics about the 
deltas and eventlists, but not the actual data). The skeleton is 
used during query planning to choose the optimal set of deltas 
and eventlists for a given query. We describe DeltaGraph in 
detail in the next section. 
The data structures are managed and maintained by several sys- 
tem components. History Manager deals with the construction of 
the DeltaGraph, plans how to execute a singlepoint or a multipoint 
snapshot query, and reads the required deltas and eventlists from the 
disk. GraphManager is responsible for managing the GraphPool 
data structure, including the overlaying of deltas and eventlists, bit 
assignment, and post-query clean up. Finally, the QueryManager 
manages the interface with the user or the application program, and 
extracts a snapshot query to be executed against the DeltaGraph. 
One of its functions is to translate any explicit references (e.g. user- 
id) from the query to the corresponding internal-id and vice-versa 
for the final result, using a lookup table. As discussed earlier, such 
a component is usually highly application-specific, and we do not 
discuss it further in this paper. 

In a distributed deployment, DeltaGraph and GraphPool are both 
partitioned across a set of machines by partitioning the node ID 
space, and assigning each partition to a separate machine (Sec- 
tion [46]). The partitioning used for storage can be different from 
that used for retrieval and processing; however, for minimizing 
wasted network communication, it would be ideal for the two par- 
titionings to be aligned so that multiple DeltaGraph partitions may 
correspond to a single GraphPool partition, but not vice versa. Snap- 
shot retrieval on each machine is independent of the others, and 
requires no network communication among those. Once the snap- 
shots are loaded into the GraphPool, any distributed programming 
framework can be used on top; we have implemented an iterative 
vertex-based message-passing system analogous to Pregel fT8] . 

For clarity, we assume a single- site deployment (i.e., no horizon- 
tal partitioning) in most of the description that follows. 

4. DELTAGRAPH: PHYSICAL STORAGE OF 
HISTORICAL GRAPH DATA 

We begin with discussing previously proposed techniques for 
supporting snapshot queries, and why they do not meet our needs. 
We then present details of our proposed DeltaGraph data structure. 

4.1 Prior Techniques and Limitations 

An optimal solution to answering snapshot retrieval queries is 
the external interval tree, presented by Arge and Vitter |3 1. Their 




Figure 3: DeltaGraphs with 4 leaves, leaf-eventlist size L, arity 
2. A Sj) denotes delta needed to construct Si from Sj. 



proposed index structure uses optimal space on disk, and supports 
updates in optimal (logarithmic) time. Segment trees |6| can also 
be used to solve this problem, but may store some intervals in a 
duplicated manner and hence use more space. Tsotras and Kange- 
laris 1 37 1 present snapshot index, an I/O optimal solution to the 
problem for transaction-time databases. Salzberg and Tsotras (27) 
also discuss two extreme approaches to supporting snapshot re- 
trieval queries, called Copy and Log approaches. In the Copy 
approach, a snapshot of the database is stored at each transaction 
state, the primary benefit being fast retrieval times; however the 
space requirements make this approach infeasible in practice. The 
other extreme approach is the Log approach, where only and all 
the changes are recorded to the database, annotated by time. While 
this approach is space-optimal and supports 0(l)-time updates (for 
transaction-time databases), answering a query may require scan- 
ning the entire list of changes and takes prohibitive amount of time. 
A mix of those two approaches, called Copy+Log, where a subset 
of the snapshots are explicitly stored, is often a better idea. 

We found these (and other prior) approaches to be insufficient 
and inflexible for our needs for several reasons. First, they do 
not efficiently support multipoint queries that we expect to be very 
commonly used in evolutionary analysis, that need to be optimized 



by avoiding duplicate reads and repeated processing of the events. 
Second, to cater to the needs of a variety of different applications, 
we need the index structure to be highly tunable, and to allow trad- 
ing off different resources and user requirements (including mem- 
ory, disk usage, and query latencies). Ideally we would also like 
to control the distribution of average snapshot retrieval times over 
the history, i.e., we should be able to reduce the retrieval times for 
more recent snapshots at the expense of increasing it for the older 
snapshots (while keeping the utilization of the other resources the 
same), or vice-versa. For achieving low latencies, the index struc- 
ture should support flexible pre-fetching of portions of the index 
into memory and should avoid processing any events that are not 
needed by the query (e.g., if only the network structure is needed, 
then we should not have to process any events pertaining to the 
node or edge attributes). Finally, we would like the index structure 
to be able to support different persistent storage options, ranging 
from a hard disk to the cloud; most of the previously proposed in- 
dex structures are optimized primarily for disks. 

4.2 DeltaGraph Overview 

Our proposed index data structure, DeltaGraph, is a directed 
graphical structure that is largely hierarchical, with the lowest level 
of the structure corresponding to equi-spaced historical snapshots 
of the network (equal spacing is not a requirement, but simplifies 
analysis). Figure[3|a) shows a simple DeltaGraph, where the nodes 
Si , . . . , 54 correspond to four historical snapshots of the graph, 
spaced L events apart. We call these nodes leaves, even though 
there are bidirectional edges between these nodes as shown in the 
figure. The interior nodes of the DeltaGraph correspond to graphs 
that are constructed from its children by applying what we call a 
differential function, denoted /(). For an interior node S p with 
children S C1 , • • • , S Ck 0we have that S p = f(S Cl . . ,5 Cfc ). The 
simplest differential function is perhaps the Intersection function. 
We discuss other differential functions in the next section. 

The graphs S p are not explicitly stored in the DeltaGraph. Rather 
we only store the delta information with the edges. Specifically, the 
directed edge from S p to S Ci is associated with a delta A(S Ci , S p ) 
which allows construction of S Ci from S p . It contains the elements 
that should be deleted from S p (i.e., S p — S Ci ) and those that should 
be added to S p (i.e., S Ci — S p ). The bidirectional edges among 
the leaves also store similar deltas; here the deltas are simply the 
eventlists (denoted Ei,E 2 , E 3 in Figure [3}, called leaf-eventlists. 
For a leaf-eventlist E, we denote by [E start , E end ) the time inter- 
val that it corresponds to. For convenience, we add a special root 
node, called super-root, at the top of the DeltaGraph that is asso- 
ciated with a null graph (Sg in Figure [3}. For convenience, we call 
the children of the super-root as roots. 

A DeltaGraph can simultaneously have multiple hierarchies that 
use different differential functions (Figure (3jt>)); this can used to 
improve query latencies at the expense of higher space requirement. 

The deltas and the leaf-eventlists are given unique ids in the Delt- 
aGraph structure, and are stored in a columnar fashion, by sepa- 
rating out the structure information from the attribute information. 
For simplicity, we assume here a separation of a delta A into three 
components: (1) A striiCt , (2) A nodeat tr, and (3) A edgeattr . For 
a leaf-eventlist E, we have an additional component, E tr ansient, 
where the transient events are stored. 

Finally, the deltas and the eventlists are stored in a persistent, dis- 
tributed key-value store, the key being (partition Jd, deltaJd, c), 
where c <G {5 struct, S no deattr, • • • , Etransient} specifies which of 
the components is being fetched or stored, and partition-id speci- 

10 We abuse the notation somewhat to let S p denote both the interior 
node and the graph corresponding to it. 




(a) Singlepoint query ti (b) Multipoint query {ti , £2 , £3 } 

Figure 4: Example plans for singlepoint and multipoint re- 
trieval on the DeltaGraph shown in Figure 3(a). 

fies the partition. Each delta or eventlist is typically partitioned and 
stored in a distributed manner. Based on the node Ad of the con- 
cerned node(s), each event, edge, node and attribute is designated 
a partition, using a hash function h p , such that, partition Jd = 
h p (nodeJd). In a set up with k distributed storage units, all the 
deltas and eventlists are likely to have k partitions each. 

4.3 Singlepoint Snapshot Queries 

Given a singlepoint snapshot query at time £1, there are many 
ways to answer it from the DeltaGraph. Let E denote the leaf- 
eventlist such that £1 G [E start , E end ) (found through a binary 
search at the leaf level). Any (directed) path from the super-root 
to the two leaves adjacent to E represents a valid solution to the 
query. Hence we can find the optimal solution by finding the path 
with the lowest weight, where the weight of an edge captures the 
cost of reading the associated delta (or the required subset of it), 
and applying it to the graph constructed so far. We approximate this 
cost by using the size of the delta retrieved as the weight. Note that, 
each edge is associated with three or four weights, corresponding 
to different attr_options. In the distributed case, we have a set of 
weights for each partition. We also add a new virtual node (node 
Sti in Figure|4ja)), and add edges to it from the adjacent leaves as 
shown in the figure. The weights associated with these two edges 
are set by estimating the portion of the leaf-eventlist E that must 
be processed to construct S tl from those leaves. 

We use the standard Dijkstra's shortest path algorithm to find the 
optimal solution for a specific singlepoint query, using the appro- 
priate weights. Although this algorithm requires us to traverse the 
entire DeltaGraph skeleton for every query, it is needed to handle 
the continuously changing DeltaGraph skeleton, especially in re- 
sponse to memory materialization (discussed below). Second, the 
weights associated with the edges are different for different queries 
and the weights are also highly skewed, so the shortest paths can 
be quite different for the same timepoint for different attr_options. 
Further, the sizes of the DeltaGraph skeletons (i.e., the number of 
nodes and edges) are usually small, even for very large histori- 
cal traces, and the running time of the shortest path algorithm is 
dwarfed by the cost of retrieving the deltas from the persistent stor- 
age. In ongoing work, we are working on developing an algorithm 
based on incrementally maintaining single source shortest paths to 
handle very large DeltaGraph skeletons. 

4.4 Multipoint Snapshot Queries 

Similarly to singlepoint snapshot queries, a multipoint snapshot 
query can be reduced to finding a Steiner tree in a weighted directed 
graph. We illustrate this through an example. Consider a multipoint 
query over three timepoints £1 , £2 , £3 over the DeltaGraph shown in 



Figure [3|a). We first identify the leaf-level eventlists that contain 
the three time points, and add virtual nodes Sti , St2, St3, shown in 
the Figure |4jb) using shaded nodes. The optimal solution to con- 
struct all three snapshots is then given by the lowest- weight Steiner 
tree that connects the super-root and the three virtual nodes (us- 
ing appropriate weights depending on the attributes that need to 
be fetched). A possible Steiner tree is depicted in the figure using 
thicker edges. As we can see, the optimal solution to the multipoint 
query does not use the optimal solutions for each of the constituent 
singlepoint queries. Finding the lowest weight Steiner tree is un- 
fortunately NP-Hard (and much harder for directed graphs vs undi- 
rected graphs), and we instead use the standard 2- approximation for 
undirected Steiner trees for that purpose. We first construct a com- 
plete undirected graph over the set of nodes comprising the root and 
the virtual nodes, with the weight of an edge between two nodes set 
to be the weight of the shortest path between them in the skeleton. 
We then compute the minimum spanning tree over this graph, and 
"unfold" it to get a Steiner tree over the original skeleton. This al- 
gorithm does not work for general directed graphs, however we can 
show that, because of the special structure of a DeltaGraph, it not 
only results in valid Steiner trees, but retains the 2-approximation 
guarantee as well. 

Aside from multipoint snapshot queries, this technique is also 
used for queries asking for a graph valid for a composite Time- 
Expression, which we currently execute by fetching the required 
snapshots into memory and then operating upon them to find the 
components that satisfy the TimeExpression. 

4.5 Memory Materialization 

For improving query latencies, some nodes in the DeltaGraph 
are typically pre-fetched and materialized in memory. In particu- 
lar, the highest levels of the DeltaGraph should be materialized, and 
further, the "rightmost" leaf (that corresponds to the current graph) 
should also be considered as materialized. The task of materializ- 
ing one or more DeltaGraph nodes is equivalent to running a single- 
point or a multipoint snapshot retrieval query, and we can use the 
algorithms discussed above for that purpose. After a node is materi- 
alized, we modify the in-memory DeltaGraph skeleton, by adding a 
directed edge with weight from the super-root to that node. Any 
further snapshot retrieval queries will automatically benefit from 
the materialization. 

The option of memory materialization enables fine-grained run- 
time control over the query latencies and the memory consumption, 
without the need to reconstruct the DeltaGraph. For instance, if 
we know that a specific analysis task may need access snapshots 
from a specific period, then we can materialize the lowest common 
ancestor of the snapshots from that period to reduce the query la- 
tencies. One extreme case is what we call total materialization, 
where all the leaves are materialized in memory. This reduces to 
the Copy+Log approach with the difference that the snapshots are 
stored in memory in an overlaid fashion (in the GraphPool). For 
mostly-growing networks (that see few deletions), such material- 
ization can be done cheaply resulting in very low query latencies. 

4.6 DeltaGraph Construction 

Besides the graph itself, represented as a list of all events in a 
chronological order, E, the DeltaGraph construction algorithm ac- 
cepts four other parameters: (1) L, the size of a leaf-level eventlist; 
(2) k, the arity of the graph; (3) /(), the differential function that 
computes a combined delta from a given set of deltas; and (4) a 
partitioning of the node ID space. The DeltaGraph is constructed 
in a bottom-up fashion, similar to how a bulkloaded B+-Tree is con- 
structed. We scan E from the beginning, creating the leaf snapshots 



and corresponding eventlists (containing L events each). When k 
of the snapshots are created, a parent interior node is constructed 
from those snapshots. Then the deltas corresponding to the edges 
are created, those snapshots are deleted, and we continue scanning 
the eventlist. 

The entire DeltaGraph can thus be constructed in a single pass 
over E, assuming sufficient memory is available. At any point dur- 
ing the construction, we may have up to k — 1 snapshots for each 
level of the DeltaGraph constructed so far. For higher values of 
k, this can lead to very high memory requirements. However, we 
use the GraphPool data structure to maintain these snapshots in an 
overlaid fashion to decrease the total memory consumption. We 
were able to scale to reasonably large graphs using this technique. 
Further scalability is achieved by making multiple passes over E, 
processing one partition (Section |4~2| in each pass. 

4.7 Extensibility 

To efficiently support specific types of queries or tasks, it is bene- 
ficial to maintain and index auxiliary information in the DeltaGraph 
and use it during query execution. We extend the DeltaGraph func- 
tionality through user-defined modules and functions for this pur- 
pose. In essence, the user can supply functions that compute aux- 
iliary information for each snapshot, that will be automatically in- 
dexed along with the original graph data. The user may also supply 
a different differential function to be used for this auxiliary infor- 
mation. The basic retrieval functionality (i.e., retrieve snapshots 
as of a specified time points) is thus naturally extended to such 
auxiliary information, which can also be loaded into memory and 
operated upon. In addition, the user may also supply functions that 
operate on the auxiliary information deltas during retrieval, that can 
be used to directly answer specific types of queries. 

The extensibility framework involves the Auxiliary Snap shot and 
Auxiliary Event structures which are similar to the graph snapshot 
and event structures, respectively. An AuxiliarySnapshot consists 
of a hashtable of string key-value pairs where as an AuxiliaryEvent 
consists of the event timestamp, a flag indicating the addition, dele- 
tion or the change of a key-value pair, and finally, a key-value pair 
itself. Given the very general nature of the auxiliary structures, it 
is possible for the consumer (programs using this API) to define a 
wide variety of graph indexing semantics whose historical indexing 
will be done automatically by the HistoryManager. For a histori- 
cal graph, any number of auxiliary indexes may be used, each one 
by extending the Auxlndex abstract class, defining the following, 
a) method Create AuxEvent that generates an AuxiliaryEvent corre- 
sponding to a plain Event, based upon the current Graph and the 
latest Auxiliary Snapshot, b) method Create AuxSnapshot, to cre- 
ate a leaf-level AuxiliarySnapshot based upon the previous Auxi- 
larySnapshot and the AuxiliaryEventList in between the two, and 
c) a method AuxDF, a differential function that computes the parent 
AuxiliarySnapshot, given a list of k AuxiliarySnapshots and corre- 
sponding k — 1 AuxEventlists. 

Any number of queries may be defined on an auxiliary index 
by extending either of the three Auxiliary Query abstract classes, 
namely, AuxHistQuery Point, AuxHistQuery Interval or AuxHistQuery, 
depending on the temporal nature of the query - point, range or 
across entire time span, respectively. An implementation of any 
of these classes involves attaching a pointer to the Auxiliarylndex 
which the queries are supposed to operate upon (in addition to the 
plain DeltaGraph index itself), and the particular query method, 
AuxQuery. While doing so, the the programmer typically makes 
use of the methods like Get AuxSnap shot provided by the Extensi- 
bility API. 

We illustrate this extensibility through an example of a subgraph 



pattern matching index. Techniques for subgraph pattern matching 
are very well studied in literature (see, e.g., fT2)). Say we want to 
support finding all instances of a node-labeled query graph (called 
pattern graph) in a node-labeled data graph. One simple way to 
efficiently support such queries is to index all paths of say length 4 
in the data graph [30]. This pattern index here takes the form of a 
key-value data structure, where a key is a quartet of labels, and the 
value is the set of all paths in the data graph over 4 nodes that match 
it. To find all matching instances for a pattern, we then decompose 
the pattern into paths of length 4 (there must be at least one such 
path in the pattern), use the index to find the sets of paths that match 
those decomposed paths, and do an appropriate join to construct the 
entire match. 

We can extend the basic DeltaGraph structure to support such 
an index as follows. The auxiliary information maintained is pre- 
cisely the pattern index at each snapshot, which is naturally stored 
in a compact fashion in the DeltaGraph (by exploiting the com- 
monalities over time). It involves implementation of the Auxln- 
dex and AuxHistQuerylnterval classes accordingly. For example, 
the CreateAuxEvent, creates an AuxEvent, defining the addition or 
deletion of a path of four nodes, by finding the effect of a plain 
Event (in terms of paths) in the context of the current graph. In- 
stead of using the standard differential function, we use one that 
achieves the following effect: a specific path containing 4 nodes, 
say (m , 712, ri3, 714), is present in the pattern index associated with 
an interior node if and only if, it is present in all the snapshots below 
that interior node. This means that, if the path is associated with 
the root, it is present throughout the history of the network. Such 
auxiliary information can now be directly used to answer subgraph 
pattern matching query to find all matching instances over the his- 
tory of the graph. We evaluated our implementation on Dataset 1 
(details in the Section|7}, and assigned labels to each node by ran- 
domly picking one from a list of ten labels. We built the index as 
described above (by indexing all paths of length 4). We were able 
to run a subgraph pattern query in 148 seconds to find all occur- 
rences of a given pattern query, returning a total of 14109 matches 
over the entire history of the network. 

This extensibility framework enables us to quickly design and 
deploy reasonable strategies for answering many different types of 
queries. We note that, for specific queries, it may be possible to de- 
sign more efficient strategies. In ongoing work, we are investigat- 
ing the issues in answering specific types of queries over historical 
graphs, including subgraph pattern matching, reachability, etc. 

5. DELTAGRAPH ANALYSIS 

Next we develop analytical models for the storage requirements, 
memory consumption, and query latencies for a DeltaGraph. 

5.1 Model of Graph Dynamics 

Let Go denote the initial graph as of time 0, and let G\e\ denote 
the graph after \E\ events. To develop the analytical models, we 
make some simplifying assumptions, the most critical being that 
we assume a constant rate of inserts or deletes. Specifically, we 
assume that a 5* fraction of the events result in an addition of an 
element to a graph (i.e., inserts), and p* fraction of the events result 
in removal of an existing element from the graph (deletes). An 
update is captured as a delete followed by an insert. Thus, we have 
that I G 1 1 1 = \G \ + \E\x8*-\E\xp*. We have that < 1, 

but not necessarily = 1 because of transient events that don't affect 
the graph size. Typically we have that 5* > p*. If p* = 0, we call 
the graph a growing-only graph. 

Note that, the above model does not require that the graph change 
at a constant rate over time. In fact, the above model (and the Delt- 



Table 2: Differential Functions 



Name 


Description 


Intersection 


J [CL, 0, C . . . J — CL 1 101 1C... 


Union 


/(a,6,c.) = aUbUc... 


OJvCWCLl 


f( n h\ — n A. r (h — a\ D<r<1 


Right Skewed 


/(a, 6) = a n b + r.(b - afl6),0<r<l 


Left Skewed 


f(a, b) = a n 6 + r.(a - an6),0<r<l 


Mixed 


/(a,6,c.) = a + n.(S a b + S bc ...) - 

T2.{pab + pbc • • • ), < r 2 < 7*1 < 1 


Balanced 


f(a,b,c.) = a + ^.(Sab + S bc ...) - 

\\Pab + Pbc - • • ) 


Empty 


/(a,6,c...) = 



aGraph structure) don't explicitly reason about time but rather only 
about the events. To reason about graph dynamics over time, we 
need a model that captures event density, i.e., number of events that 
take place over a period of time. Let g(t) denote the total number 
of events that take place from time to time t. For most real- world 
networks, we expect g(t) to be a super-linear function of t, indicat- 
ing that the rate of change over time itself increases over time. 

5.2 Differential Functions 

Recall that a differential function specifies how the snapshot cor- 
responding to an interior node should be constructed from snap- 
shots corresponding to its children. The simplest differential func- 
tion is intersection. However, for most networks, intersection does 
not lead to desirable behavior. For a growing-only graph, intersec- 
tion results in a left- skewed DeltaGraph, where the delta sizes are 
lower on the part corresponding to the older snapshots. In fact, the 
root is exactly Go for a strictly growing-only graph. 

Table|2]shows several other differential functions with better and 
tunable behavior. Let p be an interior node with children a and 6. 
Let A(a,p) and A(6,p) denote the corresponding deltas. Further, 
let 6 = a + 5ab - p a b- 

Skewed: For the two extreme cases, r — and r — 1, we have 
that /(a, b) — a and /(a, b) — b respectively. By using an ap- 
propriate value of r, we can control the sizes of the two deltas. 
For example, for r = 0.5, we get p — a + \5 a b- Here \5 a b 
means that we randomly choose half of the events that com- 
prise 5 a b (by using a hash function that maps the events to or 
1). So |A(a,p)| = \\5 a bl and |A(6,p)| = §|<U| + \p a b\. 

Balanced: This differential function, a special case of mixed, en- 
sures that the delta sizes are balanced across a and 6, i.e., 
|A(a,p)| = |A(6,p)| = \\5ab\ + \\pab\- Note that, here 
we make an assumption that a + \5 a b — \pab is a valid oper- 
ation. A problem may occur because an event G p a b may be 
selected for removal, but may not exist in a+ \5 a b- We can en- 
sure that this does not happen by using the same hash function 
for choosing both \5 a b and \p a b- 

Empty: This special case makes the DeltaGraph approach identi- 
cal to the Copy+Log approach. 
The other functions shown in Table [2] can be used to expose more 
subtle trade-offs, but our experience with these functions suggests 
that, in practice, Intersection, Union, and Mixed functions are likely 
to be sufficient for most situations. 

5.3 Space and Time Estimation Models 

Next, we develop analytical models for various quantities of in- 
terest in the DeltaGraph, including the space required to store it, 
the distribution of the delta sizes across levels, and the snapshot 



retrieval times. We focus on the Balanced and Intersection differ- 
ential functions, and omit detailed derivations for lack of space. 

We make several simplifying assumptions in the analysis be- 
low. As discussed above, we assume constant rates of inserts and 
deletes. Let L denote the leaf-eventlist size, and let E denote the 
complete eventlist corresponding to the historical trace. Thus, we 
have N — ^ + 1 leaf nodes. We denote by k the arity of the 
graph, and assume that N is a power of k (resulting in a complete 
&-ary tree). We number the DeltaGraph levels from the bottom, 
starting with 1 (i.e., the bottommost level is called the first level). 

Balanced Function: Although it appears somewhat complex, the 
Balanced differential function is the easiest to analyze. Consider 
an interior node p with k children, ci, . . . , c^. If p is at level 2 (i.e., 

ifQ's are leaves), then S p = S Cl + \S ClC2 -\p ClC2 + \5 C2C3 . 

It is easy to show that: 

|A(p,Ci)| = |(|fc 1C2 | + |p clC2 | H ), Vz 

= i(/c-l)(£*+p*)L, Vz 

The number of edges between the nodes at the first and second 
levels is N, thus the total space required by the deltas at this level 
is: N\{k - 1){8* + p*)L = \(k - 1)(S* + p*)\E\. 

If p is an interior node at level 3, then the distance between c\ 
and C2 in terms of the number of events is exactly k(5* + p*)L. 
This is because C2 contains: (1) the insert and delete events from 
ci's children that c\ does not contain, (2) (5* + p*) events that 
occur between ci's last child and C2's first child, and (3) a further 
(8* + p*)L events from its own children. Using a similar 
reasoning to above, we can see that: 

\A(p,a)\ = \{k-l)k{5*+p*)L, Mi 

Surprisingly, because the number of edges at this level is ^, the 
total space occupied by the deltas at this level is same as that at the 
first level, and this is true for the higher levels as well. 

Thus, the total amount of space required to store all the deltas 
(excluding the one from the empty super-root node to the root) is: 

( > )o **?- v > {k-l){8.+p,)\E\ 
The size of the snapshot corresponding to the root itself can be seen 
to be: | Go | + — p*)\E\ (independent of k). Although this 
may seem high, we note that the size of the current graph (G\ E \) 
is: | Go | + (£* — p*)\E\, which is larger than the size of the root. 
Further, there is a significant overlap between the two, especially if 
| Co | is large, making it relatively cheap to materialize the root. 

Finally, using the symmetry, we can show that the total weight of 
the shortest path between the root and any leaf is: \ (£* + p*) \E\, 
resulting in balanced query latencies for the snapshots (for spe- 
cific timepoints corresponding to the same leaf-eventlist, there are 
small variations because of different portions of the leaf-eventlist 
that need to be processed). 

Intersection: On the other hand, the Intersection function is much 
trickier to analyze. In fact, just calculating the size of the intersec- 
tion for a sequence of snapshots is non-trivial in the general case. 
As above, consider a graph containing \E\ events. The root of the 
DeltaGraph contains all events that were not deleted from Go dur- 
ing that event trace. We state the following analytical formulas for 
the size of the root for some special cases without full derivations, 
p* = 0: For a growing-only graph, root snapshot is exactly Go. 
8* = p*: In this case, the size of the graph remains constant (i.e., 

G\ E \ = Go). We can show that: \root\ = \Go\e l G ol . 
\root\ = lG ^l\Er 



The last two formulas both confirm our intuition that, as the total 
number of events increases, the size of the root goes to zero. Sim- 
ilar expressions can be derived for the sizes of any specific interior 
node or the deltas, by plugging in appropriate values of \E\ and 
\Go\. We omit resulting expressions for the total size of the index 
for the latter two cases. 

The Intersection function does have a highly desirable property 
that, the total weight of the shortest path between the super-root 
and any leaf, is exactly the size of that leaf. Since an interior node 
contains a subset of the events in each of its children, we only need 
to fetch the remaining events to construct the child. However, this 
means that the query latencies are typically skewed, with the older 
snapshots requiring less time to construct than the newer snapshots 
(that are typically larger). 

5.4 Discussion 

We briefly discuss the impact of different construction parame- 
ters and suggest strategies for choosing the right parameters. We 
then briefly present a qualitative comparison with interval trees, 
segment trees, and the Copy+Log approach. 

Effect of different construction parameters: The parameters in- 
volved in the construction of the DeltaGraph give it high flexibility, 
and must be chosen carefully. The optimal choice of the param- 
eters is highly dependent on the application scenario and require- 
ments. The effect of arity is easy to quantify in most cases: higher 
arity results in lower query access times, but usually much higher 
disk space utilization (even for the Balanced function, the query ac- 
cess time becomes dependent on k for a more realistic cost model 
where using a higher number of queries to fetch the same amount 
of information takes more time). Parameters such as r (for Skewed 
function) and ri,r% (for Mixed function) can be used to control the 
query retrieval times over the span of the eventlist. For instance, if 
we expect a larger number of queries to be accessing newer snap- 
shots, then we should choose higher values for these parameters. 

The choice of differential function itself is quite critical. In- 
tersection typically leads to lower disk space utilization, but also 
highly skewed query latencies that cannot be tuned except through 
memory materialization. Most other differential functions lead to 
higher disk utilization but provide better control over the query la- 
tencies. Thus if disk utilization is of paramount importance, then 
Intersection would be the preferred option, but otherwise, the Mixed 
function (with the values of n and r2 set according to the expected 
query workload) would be the recommended option. 

Fine-tuning the values of these parameters also requires knowl- 
edge of g(t), the event density over time. The analytical models 
that we have developed reason about the retrieval times for the leaf 
snapshots, but these must be weighted using g(t) to reason about 
retrieval times over time. For example, the Balanced function does 
not lead to uniform query latencies over time for graphs that show 
super- linear growth. Instead, we must choose n , r<i > 0.5 to guar- 
antee uniform query latencies over time in that case. 

Qualitative comparison with other approaches: The Copy+Log 
approach can be seen as a special case of DeltaGraph with Empty 
differential function (and arity = N). Compared to interval trees, 
DeltaGraph will almost always need more space, but its space con- 
sumption is usually lower than segment trees. Assume that + 
p* = 1 (this is the worst case for DeltaGraph). Then, for the 
Balanced function, with arity (k) = 2, the disk space required is 
0(|.E| log N). Since the number of intervals is at least \E\/2, the 
space requirements for interval trees and segment trees are 0(\E\) 
and 0(\E\ log \E\) respectively. For growing-only graphs and the 
Intersection function, we see similar behavior. In most other see- 




(b) (c) 

Figure 5: (a) Graphs at times tcurrent, t\ and £2; (b) GraphPool 
consisting of overlaid graphs; (c) GraphlD-Bit Mapping Table 



narios, we expect the total space requirement for DeltaGraph to 
be somewhere in between 0(\E\) and 0(|.E| log N), and lower if 

+ p* <C 1 (which is often the case for social networks). 

Regarding query latencies, for the Intersection function without 
any materialization, the amount of information retrieved for an- 
swering a snapshot query is exactly the size of the snapshot. Both 
interval trees and segment trees behave similarly. On the other 
hand, if the root or some of the higher levels of the DeltaGraph are 
materialized, then the query latencies could be significantly lower 
than what we can achieve with either of those approaches. For Bal- 
anced function, if the root is materialized, then the average query 
latencies are similar for the three approaches. However, for the 
Balanced function, the retrieval times do not depend on the size of 
the retrieved snapshot, unlike interval and segment trees, leading to 
more predictable and desirable behavior. Again, with materializa- 
tion, the query latencies can be brought down even further. 

6. GRAPHPOOL 

The in-memory graphs are stored in the in-memory GraphPool 
in an overlapping manner. In this section, we briefly describe the 
key ideas behind this data structure. 

Description: GraphPool maintains a single graph that is the union 
of all the active graphs including: (1) the current graph, (2) his- 
torical snapshots, and (3) materialized graphs (Figure [5]). Each 
component (node or edge), and for each attribute, each of its possi- 
ble attribute values, are associated with a bitmap string (called BM 
henceforth), used to decide which of the active graphs contain that 
component or attribute. A GraphlD-Bit mapping table is used to 
maintain the mapping of bits to different graphs. Figure[5|c) shows 
an example of such a mapping. Each historical graph that has been 
fetched is assigned two consecutive Bits, {2z, 2z + > 1. Ma- 
terialized graphs, on the other hand, are only assigned one Bit. 

Bits and 1 are reserved for the current graph membership. 
Specifically, Bit tells us whether the element belongs to the cur- 
rent graph or not. Bit 1, on the other hand, is used for elements that 
may have been recently deleted, but are not part of the DeltaGraph 
index yet. A Bit associated with a materialized graph is interpreted 
in a straightforward manner. 

Using a single bit for a historical graph misses out on a signif- 



icant optimization opportunity. Even if a historical graph differs 
from the current graph or one of the materialized graphs in only a 
few elements, we would still have to set the corresponding bit ap- 
propriately for all the elements in the graph. We can use the bit 
pair, {2i, 2% + 1}, to eliminate this step. We mark the historical 
graph as being dependent on a materialized graph (or the current 
graph) in such a case. For example, in Figure [5|c), historical snap- 
shot 35 is dependent on materialized graph 4. If Bit 2i is true, then 
the membership of an element in the historical graph is identical 
to its membership in the materialized graph (i.e., if present in one, 
then present in another). On the other hand, if Bit 2i is false, then 
Bit 2i + 1 tells us whether the element is contained in the historical 
graph or not (independent of the materialized graph). 

When a graph is pulled into the memory either in response to 
a query or for materialization, it is overlaid onto the current in- 
memory graph, edge by edge and node by node. The number of 
graphs that can be overlaid simultaneously depends primarily on 
the amount of memory required to contain the union of all the 
graphs. The bitmap size is dynamically adjusted to accommo- 
date more graphs if needed, and overall does not occupy significant 
space. The determination of whether to store a graph as being de- 
pendent on a materialized graph is made at the query time. During 
the query plan construction, we count the total number of events 
that need to be applied to the materialized graph, and if it is small 
relative to the size of the graph, then the fetched graph is marked 
as being dependent on the materialized graph. 

Updates to the Current graph: As the current graph is being up- 
dated, the DeltaGraph index is continuously updated. All the new 
events are recorded in a recent eventlist. When the eventlist reaches 
sufficient size (i.e., L), the eventlist is inserted into the index and 
the index is updated by adding appropriate edges and deltas. We 
omit further details because of lack of space. 

Clean-up of a graph from memory: When a historical graph is 
no longer needed, it needs to be cleaned. Cleaning up a graph 
is logically a reverse process of fetching it into the memory. The 
naive way would be to go through all the elements in the graph, 
and reset the appropriate bit(s), and delete the element if no bits 
are set. However the cost of doing this can be quite high. We in- 
stead perform clean-up in a lazy fashion, periodically scanning the 
GraphPool in the absence of query load, to reset the bits, and to see 
if any elements should be deleted. Also, in case the system is run- 
ning low on memory and there are one or more unneeded graphs, 
the Cleaner thread is invoked and not interrupted until the desired 
amount of memory is liberated. 

7. EMPIRICAL EVALUATION 

In this section, we present the results of a comprehensive exper- 
imental evaluation conducted to evaluate the performance of our 
prototype system, implemented in Java using the Kyoto Cabinet 
key- value store as the underlying persistent storage. The system 
provides a programmatic API including the API discussed in Sec- 
tion |3.2.1| in addition, we have implemented a Pregel-like itera- 
tive framework for distributed processing, and the subgraph pattern 
matching index presented in Section [477] 

Datasets: We used three datasets in our experimental study. 
(1) Dataset 1 is a growing-only, co-authorship network extracted 
from the DBLP dataset, with 2M edges in all. The network starts 
empty and grows over a period of seven decades. The nodes (au- 
thors) and edges (co-author relationships) are added to the network, 
and no nodes or edges are dropped. At the end, the total number 
of unique nodes present in the graph is around 330,000, and the 
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number of edges with unique end points is 1.04M. Each node was 
assigned 10 attribute key- value pairs, generated at random. 

(2) Dataset 2 is a randomly generated historical trace with Dataset 
1 as the starting snapshot, followed by 2M events where 1M edges 
added and 1M edges are deleted over time. 

(3) Dataset 3 is a randomly generated historical trace with a start- 
ing snapshot containing 10 million (10M) edges and 3M nodes 
(from a patent citation network), followed by 100M events, 50M 
edge additions and 50 M edge deletions. 

Experimental Setup: We created a partitioned index for Dataset 3 
and deployed a parallel framework for PageRank computation us- 
ing 7 machines, each with a single Amazon EC2 core and approxi- 
mately 1.4GB of available memory. Each DeltaGraph partition was 
approximately 2.2GB. Note that the index is stored in a compressed 
fashion (using built-in compression in Kyoto Cabinet). On average, 
it took us 23.8 seconds to calculate PageRank for a specific graph 
snapshot, including the snapshot retrieval time. This experiment 
illustrates the effectiveness of our framework at scalably handling 
large historical graphs. 

For the rest of the experimental study, we report results for Datasets 
1 and 2; the techniques we compare against are centralized, and 
further the cost of constructing the index makes it hard to run ex- 
periments that evaluate the effect of the construction parameters. 
Unless otherwise specified, the experiments were run on a single 
Amazon EC2 core (with 1.4GB memory). 

Comparison with other storage approaches: We begin with com- 
paring our approach with in-memory interval trees, and Copy+Log 
approach. Both of those were integrated into our system such that 
any of the approaches could be used to fetch the historical snap- 
shots into the GraphPool, and we report the time taken to do so. 

Figure|6]shows the results of our comparison between Copy+Log 
and DeltaGraph approaches for time taken to execute 25 uniformly 
spaced queries on Datasets 1 and 2. The leaf-eventlist sizes were 
chosen so that the disk storage space consumed by both the ap- 
proaches was about the same. For similar disk space constraints 
(450MB and 950MB for Dataset 1 and 2, respectively), the Delt- 
aGraph could afford a smaller size of L and hence higher number 
of snapshots than the Copy+Log approach. As we can see, the best 
DeltaGraph variation outperformed the Copy+Log approach by a 
factor of at least 4, and orders of magnitude in several cases. 

Figure ^] shows the comparison between an in-memory interval 
tree and two DeltaGraph variations: (1) with low materialization, 
(2) with all leaf nodes materialized. We compared these configura- 
tions for time taken to execute 25 queries on Dataset 2, using k = 4 
and L = 30000. We can see that both the DeltaGraph approaches 



outperform the interval tree approach, while using significantly less 
memory than the interval tree (even with total materialization). The 
largely disk-resident DeltaGraph with root's grandchildren mate- 
rialized is more than four times as fast as the regular approach, 
whereas the total materialization approach, a more fair compari- 
son, is much faster. 

We also evaluated a naive approach similar to the Log technique, 
with raw events being read from input files directly (not shown in 
graphs). The average retrieval times were worse than DeltaGraph 
by factors of 20 and 23 for Datasets 1 and 2 respectively. 

GraphPool memory consumption: Figure|8|a) shows the total (cu- 
mulative) memory consumption of the GrapnPool when a sequence 
of 100 singlepoint snapshot retrieval queries, uniformly spaced over 
the life span of the network, is executed against Datasets 1 and 2. 
By exploiting the overlap between these snapshots, the GraphPool 
is able to maintain a large number of snapshots in memory. For 
Dataset 2, if the 100 graphs were to be stored disjointly, the total 
memory consumed would be 50GB, whereas the GraphPool only 
requires about 600MB. The plot of Dataset 1 is almost a constant 
because, for this dataset, any historical snapshot is a subset of the 
current graph. The minor increase toward the end is due to the 
increase in the bitmap size, required to accommodate new queries. 

Multicore Parallelism: Figure [|Jb) shows the advantage of con- 
current query processing on a multi-core processor using a parti- 
tioned DeltaGraph approach, where we retrieve the graph parallely 
using multiple threads. We observe near-linear speedups further 
validating our parallel design. 

Multipoint queries: Figure [8lc) shows the time taken to retrieve 
multiple graphs using our multipoint query retrieval algorithm, and 
multiple invocations of the single query retrieval algorithm on Dataset 
1. The x-axis represents the number of snapshot queries, which 
were chosen to be 1 month apart. As we can see, the advantages of 
multipoint query retrieval are quite significant because of the high 
overlap between the retrieved snapshots. 

Advantages of columnar storage: Figure |8|d) shows the perfor- 
mance benefits of our columnar storage approach for Dataset 1 . As 
we can see, if we are only interested in the network structure, our 
approach can improve query latencies by more than a factor of 3. 

Bitmap penalty: We compared the penalty of using the bitmap fil- 
tering procedure in GraphPool, by doing a PageRank computation 
without and with use of bitmaps. We observed that the execution 
time increases from 1890ms to 2014ms, increase of less than 7%. 

Effect of DeltaGraph construction parameters: We measured the 
average query times and storage space consumption for different 
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values of the arity (k) and leaf-eventlist sizes (L) for Dataset 1. 
Figure |9ja) shows that with an increase in the arity of the Delta- 
Graph, the average query time decreases rapidly in the beginning, 
but quickly flattens. On the other hand, the space requirement in- 
creases in general with small exceptions when the height of the 
tree does not change with increasing arity. We omit a detailed 
discussion on the exceptions as their impact is minimal. Again, 
referring back to the discussion in Section [54] the results corrobo- 
rate our claim that higher arity leads to smaller DeltaGraph heights 
and hence smaller query times, but results in higher space require- 
ments. The effect of the leaf-eventlist size is also as expected. As 
the leaf-eventlist size increases, the total space consumption de- 
creases (since there are fewer leaves), but the query times also in- 
crease dramatically. 

Materialization: Figure [10| shows the benefits of materialization 
for a DeltaGraph and the associated cost in terms of memory, for 
Dataset 2 with arity = 4 and using the Intersection differential func- 
tion. We compared four different situations: (a) no materialization, 
(b) root materialized, (c) both children of the root materialized, and 
(d) all four grandchildren of the root materialized. The results are 
as expected - we can significantly reduce the query latencies (up to 
a factor of 8) at the expense of higher memory consumption. 

Differential functions: In Section |54} we discussed how the choice 
of an appropriate differential function can help achieve desired dis- 
tributions of retrieval times for a given network. Figure [TTJa) com- 
pares the behavior of Intersection and Balanced functions with and 
without root materialization on Dataset 1. On the growing-only 
graph, using the Intersection function results in skewed query times, 
with the larger (newer) snapshots taking longer to load. The bal- 
anced function, on the other hand, provides a more uniform access 
pattern, although the average time taken is higher. By materializ- 



ing the root node, the average becomes comparable to that of In- 
tersection, yielding a uniform access pattern. A few different con- 
figurations for the mixed function are shown in Figure [TTfb), with 
n = 0.5, r 2 = 0.5 denoting the Balanced function. As we can see, 
by choosing an appropriate differential function, we can exercise 
fine-grained control over the query retrieval times thus validating 
one of our main goals with the DeltaGraph approach. 

8. CONCLUSIONS AND FUTURE WORK 

In this paper, we presented an approach for managing historical 
data for large information networks, and for executing snapshot re- 
trieval queries on them. We presented DeltaGraph, a distributed hi- 
erarchical structure to compactly store the historical trace of a net- 
work, and GraphPool, a compact way to maintain and operate upon 
multiple graphs in memory. Our experimental evaluation shows 
that the choice of DeltaGraph is superior to the existing alterna- 
tives. We showed both analytically and empirically that the flexi- 
bility of DeltaGraph helps control the distribution of query access 
times through appropriate parameter choices at construction time, 
and memory materialization at runtime. Our experimental evalua- 
tion demonstrated the impact of many of our optimizations, includ- 
ing multi-query optimization and columnar storage. Our work so 
far has also opened up many opportunities for further work, includ- 
ing developing improved DeltaGraph construction algorithms and 
techniques for processing different types of temporal queries over 
the historical trace, that we are planning to pursue in future work. 
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